[PECOBLR-4216] Cache inline Arrow compression metadata - #1660
Draft
prathamesh-db wants to merge 3 commits into
Draft
[PECOBLR-4216] Cache inline Arrow compression metadata#1660prathamesh-db wants to merge 3 commits into
prathamesh-db wants to merge 3 commits into
Conversation
Signed-off-by: Prathamesh Baviskar <prathamesh.baviskar@databricks.com>
Signed-off-by: Prathamesh Baviskar <prathamesh.baviskar@databricks.com>
Signed-off-by: Prathamesh Baviskar <prathamesh.baviskar@databricks.com>
prathamesh-db
force-pushed
the
codex/PECOBLR-4216-cache-inline-compression-metadata
branch
from
August 26, 2026 13:53
35eb1d8 to
d20538a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Caches the compression codec from the initial Thrift inline response alongside the Arrow schema, so later result batches can omit optional result-set metadata without failing in compression detection.
If an initial Thrift response omits required metadata, the result factory now rejects it with typed
INVALID_STATEname and numeric code before dereferencing it. Inline Arrow components continue to useINLINE_CHUNK_PARSING_ERRORonce the Arrow format is known.Testing
Focused coverage includes 63 tests across the result factory, streaming and lazy inline Arrow paths, response processor, and Arrow utilities. It covers compressed and uncompressed later batches without repeated metadata and verifies the error name and numeric code for missing initial metadata.
The focused suite passed locally before the rebase, and Spotless passed on the rebased branch. CI is validating the rebased head because the local environment could not resolve dependency versions newly added on
main.Telemetry Errors
DatabricksDriverErrorCodewhere appropriate, and anynew code is uniquely numbered and tested.
requested because the author cannot access the classification.
Additional Notes to the Reviewer
Please confirm that
INLINE_CHUNK_PARSING_ERRORremains appropriate once the response is known to be inline Arrow, and thatINVALID_STATEis appropriate when metadata is absent before the format can be determined. Retry and download behavior are unchanged.